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Abstract. In this work, we introduce our approach for detecting and 
presenting anomalies of daily routines in assisted living environments. 
The proposed concept addresses the challenges of detecting, deducing 
and presenting the provenance, if such anomalies occurs. With our assis- 
tive knowledge- and model-based approach, users reach higher levels of 
awareness about their own lives and potential irregularities. Thereby, we 
consider the compelling issues of detecting causes of anomalies and also 
presenting them without overburdening the user. A prototypical imple- 
mentation substantiates our approach and introduces our forthcoming 
user study. 
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1 Introduction 


Demographic change causes many yet unsolved problems for future decades, 
such as the rise of the expenditure of healthcare or the increased burden of 
caregivers [3]. As one solution, several kinds of Ambient Assisted Living (AAL) 
systems [14] have been proposed. This kind of system aims to assist users in 
their domestic environment assuring their independence as long as possible and 
improving overall life quality [3]. 

One type of such systems features domestic living environments with activity 
recognition [18, 19]. These systems monitor surrounding environments and user 
activities in order to ensure that the elderly are living safely and independently 
in their own homes. Additionally, they can detect a range of activities, e.g., 
walking, sleeping and eating. In its core, such systems with activity recognition 
algorithms match the current available data according to sensor values. For this 
purpose, supervised and unsupervised machine learning algorithms are used to 
build and classify training and real data. 

However, beside the classification of uncorrelated activities, it becomes neces- 
sary to combine these activities to daily routines. Since a permanent interruption 
of routines could lead to diseases, ranging from sleeping troubles up to chronical 
depression [5]. Studies have shown that people, who live in regular daily rhythms, 
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Fig. 1. Anomaly Detection Workflow 


are less affected by these diseases. However, it is not always clear whether rou- 
tines differ significantly from ‘normal’ ones. The influencing parameters are not 
necessarily obvious. Also, it is not properly understood how they influence the 
daily rhythms. This poses a challenge which we solve with our procedure. We 
developed a concept in our approach, which establishes awareness about such 
deviations and present potential provenances of it. With these findings, users, 
respectively caregivers, can analyse the parameters with a corresponding trend 
to get deeper insights about their own life. 

In order to achieve the mentioned objectives, we have to answer three ques- 
tions. First, how can we detect anomalies and the potential provenance proac- 
tively within our activity recognition process? Second, how can the user ask the 
system for trends, such as sleep or training duration, due to influencing factors? 
And the third aspect is how can we present the results to the user while keeping 
the information and system interaction intuitive? Thus, we present our solutions 
to tackle the mentioned problems by adapting concepts from the area of chrono- 
biology, ambient intelligence as well as the domain of embodied assistance. 

Our test environment (and system) is a domestic living environment, equipped 
with stationary indoor sensor devices, wearables and robotic companion for in- 
teraction purposes. In our case, NetAtmo! weather stations, wearables, a smart 
phone with particular applications and a NAO robot were used (see Fig. 1). All 
parts will be outlined in detail in the following sections. The remaining parts of 
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this article consider the following aspects: First, we discuss related work in the 
fields of chronobiology, assisted living environments and embodied assistance in 
Section 2. Then, Section 3 outlines our knowledge base and clarifies how it is 
applied. Afterwards, we present the corresponding awareness concept based on 
these model in Section 4. In Section 5, we finally draw conclusions including 
ideas for future work. 


2 Related Work 


We identified multiple approaches related to our work that can be classified into 
research of chronobiology, assisted living environments as well as approaches for 
embodied assistance. However, none of these approaches combine these fields, to 
raise awareness as necessary. 


2.1 Chronobiology 


In general cases, daily routine evolve on natural rhythms of one person. The field 
of chronobiology [12] examines impacts to our biological clock, such as typical 
sleeping cycle or times of high physical fitness. 

Daily routines not only depend on personal matters, but also on exogenous 
and endogenous factors. These parameters influence the biological clock, as they 
act as orientation for the body. The most influencing example is the loss of 
daylight. For instance, experiments in a bunker showed that the loss of daylight 
extremely affects the sleep-awake rhythm. Beside the sleep irregularities, also 
diseases, such as manic-depressed behaviour, were discovered [17]. 

Another example is temperature and the exposure of light, which affect 
the overall activity of people. Changes in the atmospheric pressure can lead 
to headaches in some people and rainy conditions can reduce motivation to go 
outside. In contrast, spring time can lead to an extreme raise of activities until 
night time [12]. 

We use the findings of chronobiology for showing trends or factual knowledge 
about good life style, such as nearly eight hours of sleep or performing a ‘good’ 
amount of sport activities. 


2.2 Detecting Anomalies in Assisted Living 


The work of Yin et al. [19] illustrates an approach of activity analysis based on 
Hidden Markov Models (HMM). They combine multiple sensors (e.g., motion 
detector sensors) to HMM chains on a day-to-day basis. In contrast to other 
approaches, these models consist only of a correlation of sensor values to daily 
routines, without labelling activities. After assigning the different sensor states 
to HMM chains, the algorithm calculates a corridor of normal states. These 
reference values can be checked according to a current system state and if it is 
outside this corridor, the system can recognise an unusual and potentially critical 
situation. The advantage is that this system depends on no supervised learning. 
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Fig. 2. Semantic Training Set (Activity Module and Sensor Module) 


However, the drawback is the loss of the knowledge about the corresponding 
labeled activity. 

The work of Botia et al. [4] uses a combination of state machines, binary 
sensors and rule sets for activity recognition. At the start of the system, the 
rule set consists of only a few rules, which are refined during runtime. Every 
rule corresponds to one activity and is assigned to the binary sensors, e.g. “If 
user is in room A and no movement, then switch in state criticalSituation”. The 
user is permanently requested to solve these critical situations by correcting the 
system with a new normal activity. These corrections extend the rule set, by 
splitting old rules into new ones, “If user is in room A and is no movement 
and no presenceOfArmchairSensor, then switch in state criticalSituation”. This 
system represents a good starting point to get information about deviations and 
the provenance. But it depends too much on specific (binary) sensors and it 
has been shown that rule-based systems could lead to a high overburden of the 
user [10]. 


3 Modular Daily Routine Ontology 


The base of our activity recognition approach is a formalised, modular daily 
routine ontology called MDRO [9]. It provides a RDF-S/OWL vocabulary for 
annotating data sources, such as sensor values, factual knowledge from chrono- 
biology, recognised activities and routines. In contrast to the related works in 
Sect. 2.2, we produce a continuous link between all these factors from sensor 
values towards saved routines. 

The sensor module is a semantic description of sensors to get an unified 
view on heterogeneous sensor values and data types [11]. This means on one 
hand, physical sensors in the own surrounding, such as climatic sensors. And 
on the other hand, also uniformed, virtual sensors, such as location or sleep 
duration sensors. Additionally, we formalised healthy intervals that we acquired 
from chronobiology [12]. With these intervals it is, subsequently, possible to show 
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Fig. 3. Saved Routines with Links Fig. 4. Routine Deviation 


the users healthy trends of their own life. A good sleep duration of 8 hours or a 
hint to perform sports in evening rather than in the morning can be presented 
and thus lead to a higher awareness of influencing parameters for well being. 

The activity module holds information about possible recognised activities 
of daily living (ADL). As initial, but extendable set, we used the caring descrip- 
tions, mentioned as basic ADLs [8] and transferred them in an ordered semantic 
model of activities and sub-activities [9]. We also store the classified sensor values 
to corresponding activities in these module (Fig. 2). For instance, the classified 
activity Sleep_1 refers to the training features, which in turn link to the sensors 
and the training value. The effective detection, respectively recognition, of these 
activities is described in detail in [10]. 

The routine module saves a directed chain of ADLs with corresponding 
transition probabilities. Every routine is a chained storage of activities with 
daytime transitions to the next activity (Fig. 3). The time represents the sliding 
window for the classification, respectively the continuous stay in one activity. 
For instance, if our activity recognition algorithm uses sensor values between 
7:00 - 7:10 to determine the current activity, this combination will be stored as 
instance in the routine module and will be grouped until the activity switches. 
As shown in the illustration, the property hasActivity refers to one classified 
instance and this instance, in turn, to the semantic training set. 

The user module is utilised for authentication mechanisms and assigns all 
mentioned parts to a particular person. 

In summary, the MDRO models the concepts required for daily routine recog- 
nition. It is used to annotate sensor data, to formalise factual knowledge, to 
describe possible activities components and to save routines for users. Together, 
with these descriptions it is possible to give the user the awareness of anomalies. 


4 Monitoring Anomalies 


We split our approach in two awareness concepts. First, the proactive approach 
detects anomalies in the daily routines based on saved routines. In this proce- 
dure, the user gets information about the sensors, which have an impact on the 
deviation of the daily routine. 


Second, the reactive approach is for higher user awareness. It shows which 
sensors directly affect each other, such as sports duration and outside tempera- 
ture. And also how much they are affected and if it shows a positive or negative 
trend. 


4.1 Proactive Detection 


During runtime, the proactive detection recognises anomalies in the daily rou- 
tines, based on different time slots (e.g. week, month, quarter). Fig. 4 illustrates 
the simplified stored daily routine and the current routine that is determined 
by our Activity Recognition Algorithm [10]. The user will be notified, if the 
routine differs and is informed on which basis this difference was determined. 
For instance, if a daily routine takes an alternative route from Sleep to, e.g., Go 
For Sports way to early. This transition is used for finding all causes (such as 
reduced sleep duration in case of a to warm or to bright sleeping room). Thus 
the user can decide whether to adjust the daily routine to these new values once 
or only should be informed about the anomaly. 

To detect anomalies on the technical level, we have to group the stored graph 
and compare it to the current routine (Fig. 5). Since our routine always starts at 
0:00 and ends on 23:59, we get always the same starting and end point. Beside 
finding starting and end points, another issue was to compare different kinds of 
the same activity, if it occurs on different times. We use the same approach as 
the work from Yin [19] and group the activities in frames of six hours, to not 
overburden the user. For instance, Sleep that started between 0:00 and 6:00 will 
be grouped to the same sleep. In contrast the occurrence of Sleep between 22:00 
and 0:00 would be a non comparable instance of the same activity class. 
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Fig. 5. Routine Transformation 


After we got a comparable structure, we use an adapted VF 2 [7] graph match- 
ing algorithm, which detects isomorphic structures between the stored routine 
models and the current daily routine. We extended therefore the VF2 to not only 
detect structures, but uses the semantics in the nodes. With this approach, it is 
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possible to detect similar subroutines, even if different activities intermediates 
normal routines. The result is a map of all deviations between these two graphs. 

After getting all points of interest, we fetch the training set, respectively the 
sensor values, of previous and currently classified activity from our model. The 
sensor values are fetched from two snapshots, see Fig. 6. The snapshot (Sensor 
Snapshot) holds the current sensor values, which led to the new assumed activity. 
With the help of LIBSVM [6, 10], we get ordered weights from the classifier and 
therefore the cause why the current activity not matches the stored one. These 
weighted sensors correlate directly with the provenance of the anomaly and are 
presented to the user. 


4.2 Reactive Detection 


The reactive detection is triggered by an explicit query which can be done by the 
user for his own dataset. The user chooses the desired feature (resp. sensor), e.g., 
mood or sleep duration, and will be notified about the main factors influencing 
the chosen feature. For instance, features, that affect the sleep duration could 
result in room temperature, noise and sports duration. Beside the influencing 
factors, we implemented rules that handle whether this change is a positive or 
negative one. The foundation of these assessments is derived from chronobiology 
and formalised in our sensor module, as previously stated. As standard, but 
changeable time slot, we choose a duration of one week. 

The algorithm fetches the sensor values and firstly reduces the set of directly 
correlated sensors. This is done by the semantic (compound) description of our 
model. For instance, if sleep duration is the desired feature, it does not use sensors 
(compounds) which measure sleep directly. With this procedure, we avoid false 
positives such as, ‘sleep duration was mainly influenced by awake time’. 

To find the correlation, we use particular Factor Analysis procedure, to find 
the interdependencies between sensors. In detail we search for correlations with 
Principal Component Analysis [1] and afterwards use a Ranker to order the 
correlation. The potential influencing factors will be automatically sorted by 
this procedure. 

As this procedure gives no direct correlations in terms of percentages, we use 
the sensors with the highest probability for further processing. By the ranked 
sensors, we are able to calculate the trend of sensor values. This trend is cal- 
culated by a comparison of the last week and the health interval in our model. 
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Fig. 8. Embodied Assistance 


The combination of these factors is shown to the user and gives the awareness in 
form of: ‘Your overall sleep duration was positively influenced by increas- 
ing your sport duration this week’. It is also possible to show the influencing 
factor’s contribution rank. However, we propose that explicitly stating the con- 
tribution amount might overburden the user as the values have to be interpreted 
in the right way. Therefore, we show only the resulting sensors of the requested 
parameter, with the prior and current values. 


5 Conclusion and further work 


The increasing costs of our health care and social systems due to the changing 
population age structure in the western world pose a serious threat to our way 
of living [3]. One way to oppose these issues are domestic living environments 
with activity recognition [18, 19]. These systems monitor environments and user 
activities, which allows, for example, the elderly to stay at their homes for as long 
as possible. This can only be achieved by detecting and interpreting irregularities 
in daily routines and by providing this information to the relevant user groups, 
i.e., to family members as informal caregivers and to professional caregivers. 
Detecting these irregularities and, especially, their causes based on sensor values 
remains a challenging issue. Therefore, we have presented a knowledge-assisted 
approach to determine the derivation and showing the potential provenance to 
the user and caregiver. 

A relative new way to enhance assisted living environments is the use of 
robots within smart homes. They can assist the residents but at the same time 
providing the feeling of companionship as well [13]. While the robotic home 
companions can provide actual embodied assistance, this assistance is based on 


the data gathered by the sensors and the interaction with the robot in the smart 
home, e.g. Companionable [2] and Mobiserv [16]. 

For interacting with the system in the way we plan to use an ECA [13], once 
as a mobile application and as a real robot, see Fig. 8. However, this approach can 
be used in all situations even the NAO is not accessible, e.g., while performing 
outdoor activities. Additionally to a large extend the user can reuse learned 
interactions methods from one platform to the other, e.g., the use of the voice 
control. Next to the Embodied Conversational Agent the same information is 
displayed together with buttons to be used for touch-based input. This way, 
at least two different modalities are available for presenting information and 
getting input from the user at every time. Besides the technological aspects robot 
companions seem to have psychological benefits as well [15]. Specially the social 
companionship aspect is important which is already achieved with very simplistic 
interfaces. However, not in all cases in which such a social engagement might 
be desirable can the robot companion be physically available. Depending on the 
users abilities and desire, he can choose between voice based communication, 
text based one or a mix of both. This approach is preferable in scenarios with a 
high probability of people having impairments or even disabilities such as in the 
elderly care setting. 

Currently, we are planning to conduct an in-depth user study to measure 
the real user confidence of our approach and also determining the thresholds 
of overburden such as amount of resulting sensors in correlation detection or 
adjusting the frequency of notifying about daily routine deviations. 
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Abstract. As the number of social and psychological service robots increase, 
many robots which have emotions have been developed. In order to implement 
an emotion generation model, the authors of this paper suggest to employ the 
unified motivation theory which is proposed by Higgins. In addition, a behavior 
selection model based on emotional experience is suggested. A joint attention 
scenario is suggested in which a human and a robot play a 2048 game together 
with a follow-up interaction. A preliminary pilot test was conducted to examine 
the proposed method and it performed properly as intended. Finally, further ex- 
periment plans and expectations are discussed. 


Keywords: Motivation, Emotion, Attention, Emotional Experiences, Social 
Robot 


1 Introduction 


People expect that robots will be partners coexisting with human beings and assist 
human not only physically but also psychologically. According to World Robot Dec- 
laration which was issued by the international robot fair 2004, one of the expectation 
for next-generation robots is that robots will be social members in our human society. 
After 10 years later, nowadays, robots can be found easily in our daily lives and they 
are being used in different and various social areas such as supporting the elderly’s 
emotional communication, helping autism treatment, and so on. Social Robotics, is 
becoming one of the very important topics in the robotic field [1][2]. 

In order for robots to supply with psychological and sociable assistant, robots are 
expected to show emotional expressions. Researchers have been developed emotional 
robots, for examples, KISMET, AIBO, KOBIAN, HRP-4C, KaMERo, etc [3-7]. Re- 
cently, Pepper and Jibo were announced that they will be released in the near future, 
and got lots of people’s expectation and interest. The market of social and emotional 
robots is getting bigger. 

To design an emotional robot, the meaning of emotions for a robot needs to be 
firstly defined and a model for robot’s emotion generation should be established. 


There are several computational emotion models which already have been suggested 
[3][8-10], however, in this paper the authors in this paper will suggest a new perspec- 
tive based on the generation of the emotional state through motivations as suggested 
by the unified motivation theory. This is a novel approach respect to the current com- 
putational models of emotions. In addition, we suggest a robot’s behavior selection 
model based on emotional memory which has been experienced through interaction 
between a human and a robot. 

Motivation and emotion imply social attention [11]. If a robot is motivated in a sit- 
uation, it will concentrate and consume more resources on that situation, i.e. it shows 
more attention to the situation than other situations. Then, the robot experiences high- 
er emotional intensity according to its motivational engagement level and situational 
results [12]. Reflecting this concept to the human and robot interaction scenario, a 
robot shows more attention to social situation or human companions which are more 
emotional, and the attention and emotion affect robot’s social states such as intimacy, 
loyalty, and so on, so that the robot is able to properly behave based on the social 
states. 


2 Proposed Method 


2.1 Motivational Emotion Generation 


The definition of emotion has not yet definitely defined. The most famous psycholo- 
gist, Plutchik defined that every emotion is reaction to stimuli [13]. Although people 
sometimes seem to feel happy or angry in long time without any clear reason, psy- 
chologists have called it mood, global affect, or temperament, which are usually dis- 
tinguished from emotion. In general, emotion is defined as temporal experience and 
reaction to certain events [14]. 

According to Plutchik’s definition, the basic process of emotion generation is cog- 
nitive appraisal process, and the evaluated emotions raise physical changes and emo- 
tional reaction. As the cognitive appraisal theories describe motivation is a precursor 
of emotion, one crucial function of emotional experiences is to signal or provide 
feedback about motivational states [15]. 

Following the above concept, emotions of a robot can be defined as feedback sig- 
nals from a robot’s internal motivational states and they are caused by events. There- 
fore, motivational states and relations between motivation and emotion need to be 
organized as previous step for emotion generation. 

Motivation is like a source of behaviors, and it makes goals and encourages acting. 
Since motivations, however, cannot be observed directly, several theories exist to 
explain them: considering motivation as all-purpose energy, motivation as approach- 
ing or avoiding something, or motivation as preferences directing choices, etc. In 
order to integrate motivational theories, Tory Higgins has proposed a unified motiva- 
tion theory which describes motivation as a concept of three ways of being effective: 
value, truth, and control effectiveness [15]. Value effectiveness is about having desired 
result, truth effectiveness is about establishing what’s real, and control effectiveness is 
about managing what happens. Following this theory, in a situation that at least one of 


the effectiveness occurs, robots are motivated and engaged, then, therefore they pay 
attention and feel higher level of emotions to the situation. This motivational attention 
and emotion cause dynamic changes in robot’s behavior decision condition later on. 

According to the theory, if an effectiveness is high (success), then we experience 
pleasant emotions, such as cheerful, quiescent, confidence, vigorous, etc. If the effec- 
tiveness is low (failure), in other words, if we fail to have the desired result, we expe- 
rience painful emotions such as dejected, agitated, confused, Incompetent, etc. The 
relations between emotions and the motivational states which are value, truth, and 
control effectiveness are summarized up in Table 1. These relations are applied to the 
robot’s emotion generation model. 


Table 1. Relation between effectiveness and emtion in accordance with failure and success 


Failure (Pain) Success (Pleasure) 


Value Effectiveness 


Dejected, Agitated 


Cheerful, Quiescent 


Truth Effectiveness 


Confused, Surprised 


Confidence, Sure 


Control Effectiveness 


Powerless, Incompetent 


Vigorous, Officious 


Moreover, among pleasant/painful emotions, the emotions can be distinguished by 
levels of arousal factor which is a basic element of emotion and Regulatory focus 
theory is applied to evaluate them [12]. According to the theory, the level of arousal is 
determined by regulatory focuses: promotion focus and prevention focus. There are 
three elements that determine the regulatory focus. Those are Hedonic properties 
(positive/negative outcome situations), Standards (ideals or oughts), and Need satis- 
faction. Standards are given in the context of situation and interaction, and need satis- 
faction is depending on robot’s current conditions such as thirst, hunger, fatigue, etc. 
With these elements, promotion focus is determined by gain/non-gain situation, strong 
ideals, and nurturance needs, and prevention focus is determined by non-loss/loss 
situation, security needs, and strong oughts. After a regulatory focus is set, the level 
of arousal factor can be differentiated by the type of regulatory focus. The relations 
between the regulatory focus and the emotional elements in accordance with suc- 
cess/failure of effectiveness are summarized in Table 2. 


Table 2. Relations between the regulatory focus and the emotional elements(valence, arousal) 
in accordance with success and failure of effectiveness 


Failure Success (Pleasure) 


Promotion Focus Pain, Low Arousal Pleasure, High Arousal 


Prevention Focus Pleasure, Low Arousal 


Pain, High Arousal 


Through these procedures, we can evaluate a robot’s two emotional elements, va- 
lence (pleasure/pain) and arousal, from motivational states. With these two elements, 
we can use two-dimensional emotion space that has two axes: valence axis and arous- 
al axis. Let a robot’s emotion state be a vector a[n] in the emotion space at step n and 
let elements of the vector at step n be valence[n] and arousal[n]. Thus, the emotion 
state of a robot at step n is defined as 


a[n] = (valence[n], arousal[n]) (1) 


where valence[n] € [—1,1] and 1 denotes maximum of pleasure, -1 denotes maxi- 
mum of pain, and arousal[n] € [—1,1] and 1 denotes maximum level of arousal, -1 
denotes minimum level of arousal. 

To evaluate specific emotion type i and its intensity e;[n], we can employ Rus- 
sell’s Circumplex model [16]. The model indicates where the 28 target emotions are 
distributed in the two-dimensional emotion space. Let the position vector of the target 
emotion be t; = (valence;, arousal;) where i is an emotion type among 28 target 
emotions described in Circumplex model. The number of emotion types may narrow 
down to the emotions that are mentioned in Table 1. Then, the distance between a[n] 
and t; determines the intensity of specific emotion e;[n] by using Gaussian shape 
function as 


eiln] = exp (- (52) Q) 


where ø is a parameter for the narrowness of the Gaussian shape and it is determined 
by developers. Furthermore, we may consider the specific emotion e;[n] as an ele- 
ment of the emotion vector e[n] that contains all emotion values if it is needed. The 
authors in this paper will soon publish a paper that explains how to formulate the 
previously exposed theories as a computational model to calculate a robot’s emotional 
states. 

With these evaluated emotions, a robot is able to show not only its emotional ex- 
pressions but also reveal its internal motivation states in given situations, so that hu- 
man companion can recognize robot’s attention level and realize how much the situa- 
tions are important to the robot. 


2.2 Behavior Selection based on Emotional Experiences 


While motivational emotions are being generated, the emotional experiences are also 
memorized with the situational information. Especially, events that cause highly 
aroused emotion are well remembered [17][18]. Then, the selectively remembered 
emotional experiences affect attention and selection preference when behavior deci- 
sion is being made [19][20]. 

Likewise, a robot may store and use its emotional experiences. Since the robot’s 
emotion elements, valence and arousal, are obtained from the motivational emotion 
generation model, the robot can selectively memorize events which are attentive 
based on the arousal level. Later when the robot faces situations, it can divide its at- 
tention proportional to emotional levels that are related to the situations. The higher 
level of emotion causes more attention to the event, and the robot has preferences to 
the decision choices that match attributes of emotional experiences. The integration of 
Emotional memory and behavior decision in an autonomous robot has been recently 
tried by researchers [21-25]. 

Suppose one sample Sẹ consisting of situational states s[n,], the robot’s emotion 
a[n,], and step nę is stored when an emotion is evoked at step ng. Then, the set of all 
samples S can be described as 


S = {Si 1S; = (s[nk] alr], nk), k = 1N} (3) 


where N is the number of samples. If a situation is given as Sgiyen in the recalling 
stage, corresponding subset of samples is described as 


Ss’ = {Sk 1Sk = (Sgiven, a[Nx], Nx), Sk E s} (4) 


As explained above, the level of arousal is used as modulating factor for memory. 
The modulating factor a, can be redefined as (5) within a range [0, 1] and the modu- 
lated and re-evaluated valence v,[n] of one sample at step n can be defined as (6) 


a, = (arousal[n,] + 1)/2 (5) 
vein] = ag: valence[n,]-y(n — nx) (6) 


where y(-) is a time regression function with which more recent emotional experienc- 
es remain significant. Then, given the situation, the total accumulated valence 
Vtotailn] among S’ at step n can be evaluated by calculating 


Veotaln] = (Us: Vk [n])/M (7) 


where M is the normalization value which can be calculated as is y(n — ng). 

Now, suppose the set of robot’s action candidates for the situation is A and each 
action candidate has valence attribute v,.4ion, for example, giving the glad hand is an 
action with positive valence and avoiding is an action with negative valence. Then, 
the robot makes a decision: selecting an action that matches emotional experiences. 


action = arg Ming|Vaction — Vtotall (8) 


3 Experimental Application Scenario 


For the experiment, authors of this paper have implemented a joint attention scenario 
in which a human and a robot play a 2048 game together shown as in Fig 1. Basic 
situation is that the robot recommends a direction at every step of the game and the 
human participant press a key to move to next step in which the participant doesn’t 
need to follow the robot’s recommendation. According to the participant’s reaction, 
the robot generates its motivational emotion, and shows its emotional expressions. 
During that time the experienced emotion is simultaneously memorized with the user 
information. The game ends when the score reaches to 2048 or when there is no pos- 
sible direction. 

In this joint attention scenario, the robot has distinguished motivational states. For 
the value effectiveness, the robot is motivated to get 2048 score or higher score of the 
game than previous game scores. For the truth effectiveness, the robot is motivated to 
establish consistency of the acceptance rate of the human participant. For the control 
effectiveness, the robot is motivated to make the human follow its recommendation so 
that it can manage the game. These motivational states may differ between robots’ 
personalities or developer’s design purposes. 


2048 


~ = 


2048 


(b) 


Fig. 1. (a) 2048 game implemented in MATLAB; (b) 2048 game is displayed on a window pad 
and DARWIN-MINI is recommending a direction for next step of the game. 


Fig. 2. An interaction scenario after 2048 game in which the robot decides whether to follow 
human’s lead or not 


After the game ended, another interaction scenario is situated in which the human 
leads the robot to go somewhere by pulling the robot’s hand as shown in Fig. 6, then 
the robot decides whether to follow human’s lead or not in accordance with the ro- 
bot’s emotional experiences which were previously generated and memorized through 
previous game. 


4 Preliminary Result and Discussion 


A preliminary pilot test was conducted and examined if the proposed method would 
perform properly as intended. Figures from Fig. 3 to Fig. 7 present the preliminary 
result. 

Fig. 3 shows an example of changes of the game score and acceptance rate from 
step 1 to step 300. The game score is the biggest block number of the board, and ac- 
ceptance rate is obtained from the average rate of acceptancy in recent 30 steps. As it 
is shown, the acceptance rate is not consistent in time and it depends on how much the 
human user has intention to follow the robot’s recommendation. 


Fig. 4 shows the corresponding changes of three types of effectiveness: value, 
truth, and control. 1 denotes success of effectiveness and -1 denotes failure of effec- 
tiveness. Value effectiveness is sensitive to the game score as shown Fig. 4 (a). If the 
game score increases, i.e., the biggest block changes to higher number, the robot gets 
closer to the goal and the value effectiveness is successful in the progress. Otherwise, 
if the game score doesn’t increase, the likelihood to the ending condition of the game 
at which there is no direction to move blocks is getting higher, so that the value effec- 
tiveness is failed. Truth effectiveness depends on the consistency of acceptance rate as 
shown Fig. 4 (b). If consistency of acceptance rate of human is high, the robot can 
establish the reality of how frequently human will follow the robot’s recommendation 
or not, so the truth effectiveness is successful. Otherwise, the truth effectiveness is 
failed. Control effectiveness is sensitive to the acceptance rate by smoothing the ac- 
ceptance rate as shown in Fig. 4 (c). If the acceptance rate is higher than 0.5, then the 
robot has more authority to manage the game and the control effectiveness is success- 
ful. If the acceptance rate is lower than 0.5, the robot has lower authority so that the 
control effectiveness is failed. 

Fig. 5 shows valence and arousal factor of robot’s emotion corresponding to suc- 
cesses or failures of effectiveness. Valence can be directly evaluated by averaging 
three types of effectiveness. In Fig. 5 (b), changes of the level of arousal are depicted 
with each portion of each type of effectiveness. Though in this paper, the portion of 
each type of effectiveness is averaged, it could be weighted by predefined personality 
or could follow the winner-takes-all rule, or could affect emotion independently. 

Fig. 6 shows specific emotions e;[n] which is independently calculated by equation 
(2). For simplicity, in this paper only 8 emotions are depicted in Fig. 6, which emo- 
tions are the representative of each quadrat or axis of two-dimensional emotion space 
and picked from Circumplex model. First four emotions: pleased, exited, aroused, and 
distressed depicted in Fig. 6 (a), are more dominant emotions, Later four emotions: 
miserable, depressed, sleepy, and content depicted in Fig. 6 (b), are less dominant 
emotions in this result. During the first 50 steps, aroused and distressed emotions 
were dominantly generated because there was no significant reward from game and 
lots of uncertainty of the game. Exited and aroused emotions were dominantly gener- 
ated near 100-th step because there was a reward that is from score increase and con- 
sistent acceptance rate. During the last 80 steps, exited and aroused emotion were 
dominantly generated because there were a big score change and consistent ac- 
ceptance rate. 

As mentioned in chapter 3, after 2048 game ended, an additional interaction sce- 
nario happened. The human leads the robot to go somewhere by pulling the robot’s 
hand, then the robot decides whether to follow human’s lead or not. In the interaction 
scenario, there were four action candidates for the robot: shaking off, ignoring, hesi- 
tating, carefully following, and openly following with valence attribute -1, -0.5, 0, 
0.5, and 1, respectively. 

Assuming that the 2048 game ended at 300th step, the total accumulated valence 
Vtotal at 300th step was about 0.1 as shown in Fig. 7. As looking at the changes of the 
total accumulated valence votaj, it was negative during the first quarter of steps, and 
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3. (a) Changes of the game score which is the biggest number of the game board; 
(b) Average acceptance rate within recent 30 steps 
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Fig. 4. (a) Changes of value effectiveness; (b) Changes of truth effectiveness; 
(c) Changes of control effectiveness 
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Fig. 5. (a) Changes of valence; 
(b) Changes of arousal depicted with each portion of effectiveness 
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during the middle period, the valence value was recovered, and then finally Viotal 
reached almost to the number 0.1 although this number is not significantly big. 

At the behavior selection process, the robot decided an action which has the closest 
valence attribute, i.e. to hesitate to follow the human’s lead, which means that the 
robot still doesn’t think the human is friendly enough. The robot may sometimes care- 
fully follow the human’s lead with this Viota] by adding probabilistic parameter to the 
decision process. This probabilistic concept will be discussed in next paper. 

The reason why the value is not big enough is that the subject didn’t follow the ro- 
bot’s recommendation very much. Maybe the subject was motivated to play the game 
just for enjoying himself/herself. For next experiments, it seems to be needed to give 
reward to subjects so that they are motivated to get higher score and more to follow 
the robot’s recommendation, then, compare results caused by them to results caused 
by those who are not motivated to follow the robot’s recommendation. 

For further study, two kinds of robots will be compared: one robot acting by the 
proposed method, and the other acting randomly. Then, we will monitor participants’ 
psychological changes such as how the participants feel when the robot shows emo- 
tional expressions while playing the game, how the participants feel when the robot 
follows or disobey participants’ lead after the game, how much the participants’ ac- 
ceptance rate of robot’s recommendation changes when playing more games. 

The authors of this paper are expecting the participants to anthropomorphize the 
robot that has motivational emotions and emotional experiences and to think that the 
robot has a personality; the acceptance rate or robot’s recommendation will increase; 
and finally participants will accept the robot as a social partner. 


5 Conclusion 


In this paper, a motivational emotion generation model is proposed. The model em- 
ploys the unified motivation theory which is proposed by Higgins, and explains rela- 
tions between motivation and emotion. In addition, a behavior selection model based 
on emotional experience is suggested; with which decision preferences are more af- 
fected by highly aroused and attentive emotional memory. 

To test the proposed method, a joint attention scenario is suggested in which a hu- 
man and a robot play a 2048 game together and a follow-up interaction is conducted 
to observe the robot’s behavior selection that is influenced through the game. A pre- 
liminary pilot test was conducted and examined if the proposed method would per- 
form properly as intended. As a result, the robot showed corresponding motivational 
emotions and behaved properly based on the emotional experiences in the proposed 
framework. However, more advanced experiments are needed, and further plans and 
expectations are discussed for a comparison of a robot that has motivational emotion 
and emotional experiences and another robot which is not. 
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Beyond Patting: The Role of Attention in 
Expanding the Human-Pet Robot Relationship 


Claire D’Este 


CSIRO Autonomous Systems Program 
Castray Esplanade, Hobart 7000, Australia 


Abstract. Companion animals can provide a sense of social support to 
isolated people, including the aged, providing physiological and psycho- 
logical benefits. Although pet-like robots have been developed they have 
focused on patting response behaviours which represent only a small 
part of the human-pet relationship. In this paper we explore aspects of 
human-pet robot interactions that might benefit from the application of 
attention models; including feelings of shared experience, dependence, 
constant availability, and non-judgemental support. 


1 Introduction 


Pet ownership has consistently been shown to have positive physiological and 
psychological benefits to human health. The health savings due to pet ownership 
were estimated at $988 million for Australia for the 1994-95 financial year [7]. 
Unfortunately some of the most socially isolated people who would most ben- 
efit from a companion animal are also the most ill equipped to care for them. 
This could potentially be unhygienic for the person and life threatening for the 
animal. Australia is facing an aged care crisis with increasing numbers of vul- 
nerable elderly and decreasing numbers of aged care workers. The Australian 
department of Health and Ageing estimated that the aged care workforce will 
need to double or triple by 2050 [2]. At the CSIRO, Australia’s government re- 
search organisation, we are currently developing pet therapy robots that both 
monitor the health of aged care patients and provide social support. 

Interacting with animal-like robots has been shown to have social and psy- 
chological benefits [19] and foster social interaction [11] in aged care patients. 
The robot interactions have been limited to simple expressions and sounds trig- 
gered by touch. In recent models they have included the ability to respond to 
the robot’s given name. Studies have not been conducted to show if these inter- 
actions are enough to foster long term engagement and therefore benefit. There 
is also ethical concern that these robots have a possible infantilisation effect on 
aged care patients [17] and that any suggestion that it is a true relationship is a 
deception. 

Studies with companion robots for the elderly have only scratched the sur- 
face in simulating pet relationships. In this paper we explore the human-pet 
relationship in more detail and the role of attention in reproducing the neces- 
sary behaviours. 
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2 The Human-Pet Relationship 


Married or co-habiting couples demonstrate stronger health and longer life ex- 
pectancy than those who are divorced or single [12]. Animals have an important 
role in the lives of people in transition; for example the newly married, divorced 
or widowed [14]. A lack of social support is a significant risk factor for subse- 
quent physical and psychological problems [18]. Companion animals are a social 
support in and of themselves, but also act as facilitators of social interactions 
between other human beings [3]. Beck and Katcher [3] suggest that sharing our 
lives with companion animals promotes a sense of safety and consistency. 


“by encouraging touch and giving humans a loving creature to care for, 
the interaction with animals stimulates physical reactions that are very 
necessary and important in humans. [4]” 


To expand the relationship with pet robots beyond their responding to touch 
we need to investigate what it means to be a ‘loving creature for someone to 
care for’. Attachment theory suggests that humans have a need to protect and 
be protected [1]. Pets depend on humans as children rely on parents for their 
continual care, protection from danger, and explanation on their behalf because 
of their lack of language. Further to this many older pet owners regard their pet 
as a symbol of independence [6]. This symbol allows them to manage the process 
of becoming increasingly dependent on other humans for their needs; including 
financial. The pet is dependent on them, and therefore they are not at the lowest 
step on the social hierarchy. This ‘need to be needed’ can manifest itself even 
more strongly giving a socially isolated person a reason to get up in the morning 
and a reason to take care of themselves effectively [20]. 

As social supports pets may reduce loneliness and contribute to a general 
sense of well-being in their owners [16]. Study results report that as social sup- 
ports their benefits include their [5]: 


— constant availability 
— non-judgemental support 
— unconditional love 


This sense of non-judgemental support may be explained by pets appearing 
to be indifferent to our physical appearance, social status, material possessions, 
relative intelligence, and conversation skills. People that feel rejected, a burden, 
or invisible to society may find the relationship and support they need in an 
animal. 

Simulating unconditional love may be a controversial prospect at best, but 
many of the other perceived advantages might benefit from the application of 
attention models; including feelings of shared experience, dependence, constant 
availability, and non-judgemental support. 
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3 Attention 


3.1 Shared Experience 


Shared experience is an important aspect of the feeling of sharing your life with 
another. Kaplan and Hafner [10] describe joint attention as a shared intentional 
relation to the world. This extends beyond ‘simultaneous looking’ and is an 
active bilateral process. Joint attention is intentionally directed perception that 
involves one or more parties trying to achieve a goal. Kaplan and Hafner [10] 
specify four stages of joint attention that will be used to describe pet robot 
requirements: 


— Attention detection - eg. tracking the other agent’s behaviour and gaze. 

— Attention manipulation - eg. pointing gestures and words. 

— Social coordination - eg. turn-taking and role-switching. 

— Intentional stance - eg. interpreting other’s intentions, including those 
different from one’s own. 


Yonezawa et. al [21] concluded from experiments with a stuffed toy robot that 
joint attention affected human interactors positively only when combined with 
methods for maintaining eye contact. This is likely also applicable to pet-like 
interactions. 

Shared experience can involve shared response to abnormal events. These 
events might include loud noises, new visitors, or extreme weather conditions. 
These in turn could trigger curiosity behaviours, or simulating appropriate re- 
sponses, such as heat seeking in cold weather. This could be implemented in 
a ‘simultaneous looking’ sense in which both parties simply react to similar 
events. Reliable attention detection might allow the robot to respond to events 
that it cannot detect itself. However, including social coordination and inten- 
tional stance behaviours to involve an interpretation of the owner’s reaction to 
the event might elevate this to a shared experience. For example, behaviours 
that might be interpreted as ‘Did you hear that?’ and Should I be concerned?’. 

Habituation and sensitisation models allow an agent to autonomously adapt 
to its environment. 


“Individuals do not receive a positive or negative feedback for their 
responses from the environment. Instead, they measure the effectiveness 
of their responses based on their internal judgement of the significance 
of the repeated stimulus.” [13] 


However, to create shared experience it may be necessary that the dominant 
party (the pet owner) guides habituation and sensitisation in the robot. Humans 
and pets are not social equals, and pets are expected to learn our rules and ways 
of living. The pet must look to the human to determine the appropriate response 
to events; both normal and abnormal. 
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3.2 Dependence 


The goal for much of robotics is to increase their autonomy to the stage where 
human intervention is not required. In the case of the human-pet robot relation- 
ship dependence may be a necessary requirement. 

Attention manipulation is a candidate for providing the sense of the depen- 
dence of the pet robot deemed necessary for attachment. The robot should be 
able to direct the human to any needs or fears it might have until they are 
appropriately met. 

A simple system to implement would be to require the owner to charge the 
robot’s batteries. The robot then ceases to function when the owner neglects it. 
This does not have the same responsibility as not feeding a living creature, as 
the robot would be essentially unharmed if allowed to run flat. Potentially the 
robot could lose any personalisation it has acquired to provide incentive. 

Another approach would be to foster empathy in the owner that increases 
their desire to keep the robot safe. Imitation has been shown to be linked with 
empathy [8]. Social coordination is required to play imitation games that could 
assist in this process. 


3.3 Constant Availability 


It is a simple matter to program a robot to be always present at the owners 
home. However to appear ‘constantly available’ there are technical challenges 
in reliability to consider. The robot would need to be technically robust and 
easy to keep charged. In addition the robot would need to reliably find and 
provide attention for the owner. It is unlikely that pet owners are referring to 
the availability as an inanimate object might have, in that they can lay their 
hands on it at any time, but more that the attention is there any time they 
need it. The robot should then have a system that accurately sense when their 
attention is required and provide it at a priority to other tasks. In practice this 
may involve sensing when the owners social needs are not being met and make 
itself available for patting or company. 

It is suggested that animals feel empathy as they appear to be affected by 
the emotional distress of others. Animals may become agitated or distressed by 
anothers distress, and act to eliminate the stressor or soothe them [15]. The robot 
actively making itself available when the owner is distressed may extend this to 
a more animal-like interaction. At its simplest this is an attention detection 
problem, but emotional events may be harder to recognise than physical ones. 


3.4 Non-Judgemental Support 


Because of the requirement to program perceptive and discriminatory behaviours 
into our robots, in a sense we gain ‘’non-judgemental support’ for free. The 
robot will only pay attention to the characteristics of their owners that they 
are explicitly told to. What we program the robot not to pay attention to may 
be as beneficial as what we do. For robots that are also acting as physiological 
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and psychological health monitors it may be important to track and report on 
these without appearing to be disapproving. It may be that animals are able 
to express concern, interest, and attention in a manner that is non-invasive and 
non-patronising. One of the design decisions to be made in a health monitoring 
robot is whether it should be verbal or non-verbal. 

Reproducing the non-verbal attention of pets may be necessary to generate 
feelings of non-judgemental support. Puppies that have had little to do with 
humans are able to follow gaze and pointing gestures, whereas wolves cannot. 
This suggests the process of domestication has resulted in skills in interpreting 
human social cues. Dogs themselves use gaze, body orientation, and pointing 
cues to establish joint attention [9]. 


4 Conclusion 


Pets provide us with a relationship that extends far beyond the affection we give 
and they receive. Joint attention methods such as attention direction, social 
coordination, and intentional stance may assist in elevating these interactions to 
the sense of a shared life and social support. 
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1 Introduction 


On June 5*” 2014, a robot named Pepper, hailed as emotionally responsive and 
meant solely to emotionally interact with human beings, was unveiled in Japan. 
“Our aim is to develop affectionate robots that can make people smile” [12] stated 
Softbank CEO Masayoshi Son. But what did Masayoshi mean the robot would 
“make people smile?” Did he mean the robots behavior might elicit a smile 
from a human? If so, some people smile when they are angry in order to make 
themselves less aggressive in a tense situation. Certainly, he did not mean he 
developed a robot to make people angry! So we may assume Masayoshi meant 
human interaction with Pepper would make people “happy”. However, by what 
methodology do we determine the robot created happiness in us? The viewpoint 
of this essay is that the humanities and specifically the performing arts are 
in an excellent position to develop a model of behavior between humans and 
robots in order to present a clear, authentic emotional relationship between 
them. Meaning, the social intelligence of a robot interacting with a human is 
akin to the dramatic relationship between human beings while performing for 
an audience. 

The stage is the social arena and the performance styles are the tools with 
which to sculpt a meaningful performance. We must begin to understand how 
we perform for each other all the time as we are essentially building robots to 
perform for us. This is not an entirely new concept. Erving Goffman, notable 
sociologist used the metaphor of the stage to dramatize his views of how human 
social interaction is a series of performances that we tweak and mold instanta- 
neously both inter- and intra-personally. 


“Within the walls of a social establishment we find a team of performers 
who cooperate to present to an audience a given definition of the situa- 
tion. This will include the conception of own team and of audience and 
assumptions concerning the ethos that is to be maintained by rules of 
politeness and decorum” [5]. 


Goffman discusses the presentation of self through a social situation in which we 
often try to cooperate with our fellow social actors. “Typically, but not always, 
agreement is stressed and opposition is underplayed”|5|, yet the expression of 
emotion in human beings does not always follow such agreeable rules. Goffman 


wrote specifically how the individual compartmentalizes his responses into a 
“front” of the house -the actor and performer- and the “back” of the house -the 
performers true identity [5]. 

This essay seeks to present a system to express deeper emotional authen- 
ticity through the utilization of performance behavior, gesture, shape, and re- 
lationship. This is accomplished through the implementation of traditional per- 
formance techniques re-contextualized from a human/human stage model to a 
human/robot social intelligence model. 


2 Social Intelligence, Attention and Authenticity 


When analyzing social intelligence between robots and humans, we must be cog- 
nizant of the complexity and specificity of emotional connections human beings 
share with each other in the multitude of everyday moments of interaction. Why 
is the artist involved in this decision? Performers study an art form nearly 3,000 
years old that demands a level of authenticity and skill from a collaborative team 
including actor, director, and playwright. There are few art forms or disciplines 
as adept as that of the theatre artist to convey how to create, hold, build, and 
challenge attention from a group of people -the audience- from whom no pre- 
vious relationship was originally held. And yet, social intelligence, by definition, 
strives to do just that. Drama and the performing arts, specifically well-known 
performance techniques, can be used to understand, deepen, and clarify emo- 
tions, as well as, focus attention in a social relationship not only through human 
connection, but by expanding performance techniques to relate to human beings 
interacting with robots and vice versa. 

Many visionaries have used the medium of drama and performance to ex- 
press their artistic ideas. Constantin Stanislavski (1863-1938) emphasized natu- 
ral, (re)created, emotional expression in response to the arguably more superfi- 
cial theatre of the time. “The approach we have chosen —the art of living a part- 
rebels with all the strength it can muster against those other current ‘principles’ 
of acting. We assert the contrary principle that the main factor in any form of 
creativeness is the life of a human spirit, that of the actor and his part, their 
joint feelings and subconscious creation” [9]. Theatre director/actor Stanislavski 
is widely considered the ‘father’ of modern theatre and his Stanislavski system 
was the framework for “the method” which was refined and popularized by Lee 
Strasberg in the U.S. 

When creating successful drama in the theatre arts we speak about specifics 
of a character through their behavior, their gesture, their shape and the interac- 
tion of those attributes functioning through the very specific relationship implied 
between the characters. In order to make sense of behavior, gesture, shape, and 
relationship, the artist must determine the given circumstances of the play. 

These technical terms are defined as follows: 


Given circumstances: the givens of the moment in a play. Essentially, where 
a character finds themselves (physically and emotionally) at that moment 


in their world and how they respond to people and events and unfolding 
around them because of those givens. 

Behavior: How a character behaves physically and emotionally in order to ap- 
proach the given circumstances of the specific moment. For example, a char- 
acter may get down on bended knee to request a proposal of marriage while 
another drags their partner to the nearest justice of the peace. 

Gesture: The movements a character uses within their characters behavior and 
for what specific effect. For example, take a simple gesture of holding or 
shaking hands. What variety can be expressed in this simple gestures and 
what is the meaning in the choice to serve the relationship? 

Relationship: What are the parameters of the person with who we share the 
stage or social arena? For example, are they a peer or subordinate? We speak 
and act differently to the police officer that has stopped us for speeding than 
we do to the teenager that has come home past their curfew. 

Shape: What is the overall tone of the character and genre the play is trying 
to express? Can we describe it as comedy, drama, dramedy? 


In regards to all the defined terms above, it is assumed that a deepening 
in the understanding of what the given circumstances are at any one moment 
culminates in a heighted state of social intelligence. The ability to be cognizant of 
our social attention skills can be trained; however, is not necessarily training for 
an artist only. It can be training for an engineer or a roboticist. It is training to 
explore the nuances of how attention is manifested in social intelligence through 
the application of performance. One type of attention necessary to understand 
the given circumstances of social interaction would be the exploration of sensory 
data. An awareness of what we see, hear, smell, feel kinesthetically both inside 
our bodies and the tactile experiences on the surface of our bodies, as well as 
taste. An artist develops an ability to create a relaxation in their body and 
thereby concentrate the stream of sensory impulses occurring in order to focus, 
to prioritize, or to give attention to the given circumstances the sensory data 
refers to. At the moment the sensory data is received, the will of the artist 
engages. The will is what determines the course of action given the stream of 
sensory data. It is not enough to simply feel, the will guides the artist in their 
dramatic objectives and choices. In a social intelligence arena with human robot 
interaction, the will is the determination that guides attention. In a performance, 
attention is greatly affected by sensory data culled from the individuals memory, 
which then affects the artist emotionally. 

Yet, the training of sensory impulses and will are not meant to create actors 
out of an audience of lay people, they are meant to specify and analyze human 
behavior —which we are all familiar with- from the performative point of view of 
the artist and now applied to human-robot interaction. The social intelligence 
of a character in the given circumstances of the current scene, the inter-relating 
with other characters or the audience, and ultimately how or why the character 
behaves, is the research now transcribed to the relationship between humans 
and robots. The very best conclusion for human-robot interaction would con- 
tain not only an authentic sense of how humans behave between themselves in 


relationship-based dramatic circumstances -theatre arts- but also how a human 
may interpret and thereby behave with a robot with whom they may begin to 
have very human-like emotional relationships with. 


3 The Building Blocks of Authenticity 


3.1 Toward Performative Arts 


If we are to look toward the artist for answers in how humans respond to hu- 
mans and ultimately how humans may respond with robots, there is a need to 
research the many technicians, directors, and artistic visionaries, who created 
performance styles for the performing artist. These performance styles present 
an exploration of behavior, gesture, shape and relationship as it relates to how 
human beings interact and how robots must behave if human beings are to 
authentically interact with them. Research points toward the efficacy of long- 
standing performance techniques or styles, styles that has survived through not 
only decades but centuries of performance. The preference to present two dif- 
ferent, yet mutually compatible, framing mechanisms to illustrate how human 
beings may interact with our robot companions, is a helpful aspect. 

The first technique presented is Vsevolod Meyerhold’s biomechanics [1]. Mey- 
erhold’s biomechanics is a performance technique that stresses external physi- 
calization, gestures, behavior, and avant-garde themes and movement to create 
emotion and authentic characterization in the performer. It is representational, 
abstract, and features a theatrical expression known as the grotesque. Biome- 
chanics is an external performance technique created, developed, and best exem- 
plified by Meyerhold with his work in Russia from the 1890s to the present. Life 
of Man by Leonid Andreev with its exemplary lighting design is one example of 
Meyerhold’s innovative work from this time [1]. 

Although contemporaries, Meyerhold’s techniques were quite different from 
the second presented performance technique: Lee Strasberg’s method [10]. Orig- 
inally developed from Constantin Stanislavski’s techniques, Strasberg’s method 
technique stresses thoughts, psychological motivations and desires, personal mem- 
ories, and naturalistic behavior and movement to create emotive characteriza- 
tions in the performer. Internal performance technique was created, developed, 
and best exemplified by Lee Strasberg and his work to create the method system 
in the United States from the 1920s to the present. Most notable of Lee Stras- 
berg’s work was the co-creation of the legendary yet short-lived, The Group 
Theatre, and specifically the production directed by Strasberg of Golden Boy 
by Clifford Odets produced to notable acclaim both on Broadway and in Lon- 
don [3]. 

This research utilizes aspects of both contrasting performance techniques to 
fully develop intense and effective emotive bond between humans, and now as 
technology is creating emotional robots, between humans and robots. 


1 The grotesque theatre as defined by Bert Cardullo and Robert Knopf is simply “the 
resolving of tragic situations into comedy or the reverse” [2] 


The function of a performance technique is to deepen the authenticity of the 
performer so the relationship between the player and the audience is more im- 
mersive and ultimately more satisfying. We go to the theatre to feel something 
—a human connection, an understanding of why this human being, in these cir- 
cumstances, made these choices. That understanding is similarly translatable to 
a relationship between a robot and a human, in fact it is wanted and desired. 
Attention for social intelligence will be most effective if it stresses immersive 
emotional and relational bonds. As robots move into the social arena of com- 
panions to humans, it is imperative to understand and create relationships on a 
sincere and accessible performance level. 

The proposed performative methods might be an innovative and interesting 
way to elucidate a unique insight to the authentic emotional responses required 
when working with actors, digital characters (videogame characters), animators, 
and now, engineers and roboticists. 


3.2 When Emotions are more than Right or Wrong 


When speaking of emotion, a significant caveat is that although there may be 
many different ways to express emotion, there are very few emotions to express. 
Some psychological studies theorized a model of emotions based on only a lim- 
ited number of primary emotions. Paul Ekman theorizes six basic emotions [4] 
while a recent study conducted at Glasgow University now proposes only four 
[7]. Yet, through this limited landscape the artist seems fluent in a nearly infinite 
variety of expression across a landscape of social, cultural, and geographic fram- 
ing structures. One method for accessing emotional authenticity in performance 
is to eliminate a judgment of right or wrong in defining the emotion. 

There is a common tendency to assign emotion as right or wrong. Anger may 
be considered a bad emotion while courage is good. However, research consistently 
highlights that authentic, expressive human emotion relies very little on the right 
or the normal relationship. This tendency has been observed in sociological 
studies made by MIT professor, Sherry Turkle. Turkle introduced a study she 
conducted with children and a robot named Cog [11]. In her analysis, Turkle 
notices the children enjoying their interaction with Cog. The children played with 
the robot and seemed to nurture it and care for it. “They treated it as a creature 
with needs, interests, and a sense of humor” [11]. The children seemed quite 
bonded to the robot and accepted Cog as a playmate as they would a human 
child or peer with all the usual emotional responses that human beings reserve for 
their human counterparts. The children’s behavior implied that Cog was not less 
of a playmate because Cog was not human. This summation was not completely 
unexpected by Turkle who has cited Joseph Weizenbaum’s work regarding the 
Eliza Effect and its strong emotional influence on its participants |11]. However, 
Turkle also planned a meeting with Cog where each child was systematically 
shown Cog’s inner mechanisms and working parts. Cog was exposed as plastic, 
metal, and digital instructions. The childrens relationship with Cog was not 
normal in any socially conforming version of a typical relationship, yet none of 
this mattered. The children again quickly related to Cog in the same nurturing, 


relational manner as before the nuts and bolts explanation. Nurturance is the 
relationship in this given circumstances that drove the children’s emotions to 
care for Cog. The children cared for Cog the same way they seem to care for 
and feel for each other. Even though, as Turkle laments, “To have a relationship, 
the issue is not only what the human feels but what the robot feels” [11]. Turkle 
summarized the study stating nurturance should be known as the “killer app” 
[11], creating a bond between humans and non-organic objects that is incredibly 
strong. Therefore, although cultural norms of right, wrong, or normal may be 
recognized, they are secondary to the authenticity and creation of emotion, and 
the relationship that may form because of that creation of emotion. 


3.3 Human-Robot Mutual Need: an Analogy with Drama 


One way to deepen the relationship between humans and robots is to note that 
human relationships are often based on mutual need. Drama is a series of ever 
unfolding conflicts continually testing the bounds of which character’s need is 
currently being fulfilled. Humans respond favorably to the act that helps another 
while it also fills a need for them. This inter-relationship is being explored in a 
human-robot relationship. For example, Okada Michio developed a robot that 
does not pick up trash, but rather finds it and requests a human’s help to accom- 
plish its task [6]. In this instance, when the robot finds a piece of litter, it does 
not function autonomously, but rather indicates the location of the litter and 
with an angled bow, requests the help of a human to pick it up. The implication 
is clear; let’s work together and see what we can accomplish. Okada believes this 
relationship creates a desire to nurture the robot echoing Sherry Turkle’s study. 


3.4 The Perfection of Imperfections 


Another method of building a relationship may be in the creation of a robot 
whose characteristics are imperfect. Just an in human beings, some of the most 
unique and charming human characteristics are those that are seemingly imper- 
fect -a gap tooth, a cowlick in the hair- an imperfect robot might be one that 
stutters occasionally, or develops a limp after moving for an extended period of 
time. As illustrated by the uncanny valley? research, human engagement is not 
in what is perfect, but rather in the abstract or the imperfect. Imperfection in 
all its unique qualities is endearing to humans. It would follow that we may like 
our robots to be more imperfect in order to bond emotionally with them. 


? The uncanny valley was a term originally coined by Masahiro Mori in 1970 to de- 
scribe the eerie and unsettling response of people when confronted with an android 
that is not quite human. The uncanny arises when one is confronted with something 
that transgresses the boundaries of our conceptions of normal, and thus seems to be 
one of the key responses invoked in confrontations with the border between humans 
and machines [8] 


4 Conclusions 


When developing robots that will help us in our day-to-day lives should we be 
looking at the mutual need humans and robots may begin to feel for each other? 
How will those relationships be expressed? How can we develop robots that spur 
our desire to connect with them in order to fulfill the individual’s wishes and 
needs? Will a robot ever have a need for a human’s emotional attention? These 
are all relevant questions for future research. 

Given the proposition that the analysis of the human condition through per- 
formance is helpful, and even essential, in the development of social, attentive, 
emotional, and empathetic human beings, are performance techniques then es- 
sential to analysing the behavior and relationship between humans and robots 
in the arena of social intelligence? 

Performance, connection, and authenticity are the gold standard that should 
be hoped for in the creation of the relationship between humans and robots. We 
can attain these goals through the specific application of understanding human 
behavior, gesture, shape and relationships. We must be specific in programming 
robots to respond to, and to eventually to learn from, the human relationships 
they will be a part of. Social intelligence and the attention to create, improve, 
and expand the understanding between humans and robots can be positively 
impacted by the research of performance styles and techniques, specifically in- 
ternalized method work, and representational biomechanics. 
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Extended Abstract 


Focusing on the interactiviness that a robotic interface establishes between the virtual 
and the real world, some sensory systems and mobile robotic platforms were devel- 
oped for the AURAL project, a robotic evolutionary environment for sound produc- 
tion [1, 2]. 

In the AURAL environment, the sonification is generated by an evolutionary map- 
ping of the robot trajectories into sound events. One issue addressed is the struc- 
ture/novelty tradeoff [3]. By applying these ideas to algorithmic composition systems 
means that more knowledge and structure allows the creation of new pieces that are 
more tightly matched to the desired musical genre. However, the flipside of more 
structure is less new material. The highly constrained output will be less likely to 
stray beyond a genre’s limitations or it may be surprising. Thus, the highly structured 
composition system will be less general, able to reach less ‘music space’ with its out- 
put. 

In the AURAL, this tradeoff is treated by creating an interplay between sound, re- 
al-world artifacts, user and behavioral information. Through the interaction among the 
evolutionary sound process, the artificial vision system and the mobile robots [4]. The 
sound interface has a Graphic Area, the heart of the system, wherein the user may 
draw curves to be sent as trajectories to the robots. This area is associated with a con- 
ceptual sound space with two axis, the “red” one, or melodic, and the “blue” one, or 
rhythmic. The paths travelled by the robots in the arena are observed by the artificial 
vision system and sent, as sequences of points, to the sonification module. The red 
curves, sent as trajectories to the robots and the blue curves associated with the paths 
travelled guide the evolutionary sound process across different regions in the sound 
space. 

AURAL was presented in an art gallery where the visitors could appreciate the 
sound output and the interaction among the robots, as a kind of choreography. The 
visitors drew curves in the graphic area, which were transmitted as trajectories to a 
master robot, the Nomad. While the robots (until 4) moved in the arena, virtually 
traveling along the conceptual sound space, people changed the orchestra, rhythm and 


pitch controls, investigating the sound possibilities. Both a process of man-machine 
interaction and parallel exploration occurred. 

On the last day of the exhibition, a dancer, three musicians and the AURAL system 
itself, with four robots, performed an interactive concert called Robotic Variations. 
The performance of the robots was accompanied by the musicians, who knew the type 
of music that would be generated, but they had to be able to adapt the performance. 
At the same time, the dancer, tracked by her red hat, was interacting with the robots, 
all interfering in the music that was being generated. 

One can see as an interesting aspect of the AURAL environment the possibility of 
different setups to explore distinct levels of interaction among humans and machines. 
AURAL supplies a platform for robotic experimentation and artistic creation explor- 
ing human and machine interaction and bringing about results that could not be ob- 
tained otherwise. 
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Extended Abstract 


An artificial vision system, mobile robots and a generative process were applied for 
sound production in AURAL), a robotic art installation [1]. Generative systems have 
many similarities with systems found in various areas of science; they may provide 
order and disorder, as well as a varying degree of complexity, making behavioral 
prediction difficult. However, such systems still contain a definite relation between 
cause and effect. The artist (or creator) generally provides basic rules, and then de- 
fines a process, random or semi-random, to work on these elements. The results con- 
tinue to happen within the limits domain of the rules, but also may be subjected to 
subtle changes or even surprises [2]. 

In the AURAL), synthetic, game and everyday sound fragments are inserted into a 
database, the memory of the system. Each sound fragment is associated with a cell in 
a virtual grid, projected on a winding format platform (3m x 3m wide, 0.3m high), or 
stage (Figure 1). A hole inside the platform creates tracks that may be travelled on by 
only one robot or two robots. The robots have a border sensor, they stop when they 
detect the border. In the other regions of the platform, three or four robots can move 
around. This design cases conflicts among the robots when they try to escape from 
confined areas. The robots are tracked by a vision system which evaluates the position 
(x, y) of the robots on the stage; associates a cell in the matrix with that position and 
plays the sound fragment associated with it. The movement of the robots through the 
different regions of the stage triggers the sound of the associated cells, (re)creating 
soundscapes in the installation environment. 

On a TV, the virtual grid is shown in several angles, as well as the cells activated 
by the robots (Figure 6). The visitors may interact with the system by talking, singing 
or screaming at a microphone, starting the intervention process: sound fragments are 
extracted from the interventions of the visitors and randomly inserted into the envi- 
ronment matrix; there is a possibility of the segments to be triggered and played again 
by the movement of the robots. A spectral analysis is applied on the fragment that 
caused the intervention, and two visual effects may be perceived by the visitors. 


When there is more energy in the upper partials of the sound fragment, the following 
actions take place: the color of the cell associated with that fragment is changed to red 
on the TV, otherwise to blue. A rotation is applied on the grid. Finally, the sound 
fragment is inserted into the sound data base, i. e., the memory of the system, super- 
posing a previous one, enhancing a recycling acoustic process. Thus the 
AURAL), intervenes in the sound ecology, generating new aural trajectories with the 
everyday sounds. 


Fig. 1 The components of AURAL,: sound database, camera, microphone, robots. 


Acknowledgements 


We thank the students Felipe Shida and Daniel Kantor. We thank the Scientific Initia- 
tion Program of the National Research Council (PIBIC/CNPq), the Center for Tech- 
nology Information Renato Archer and the Interdisciplinary Nucleus for Sound Stud- 
ies of the State University of Campinas (NICS/UNICAMP) for making this research 
possible. We thank the Serviço Social do Comércio (SESC) for its sponsorship and 
opportunity of presenting this work at SESC-Campinas gallery. This research work 
was part of the AURAL project, supported by the Foundation for the Research in Sao 
Paulo State (FAPESP) process 05/56186-9. Manzolli is supported by the Brazilian 
Agency CNPq. This work is licensed under the Creative Commons [CC-BY]. 


References 


1. Moroni, A., Manzolli, J.: AURAL). In: INSTANTE, Arte e Tecnologia no SESC Campi- 
nas, Serviço Social do Comércio, 2011. 

2. Moroni, A., Manzolli, J.: Multimodal Robotic Artistic Installations for Sound Production. 
In: Anais do IV Seminario de Musica, Ciéncia e Tecnologia, 2012, Sao Paulo. 


COLUMN: Core Less Unformed Machine 


Yasutaka Takeda, Kohei Yoshida, Shotaro Baba, P. Ravindra S De Silva, and 
Michio Okada 


Interactions and Communication Design Lab, Toyohashi University of Technology 
Toyohashi 441-8580, JAPAN 
{takeda, yoshida, baba}@icd.cs.tut.ac.jp, {ravi, okada}@tut.jp 
http://www.icd.cs.tut.ac.jp/en/profile.html 


1 Introduction 


Fig. 1. Appearance of Core Less Unformed Machine (COLUMN). 


The concept of a self-configurable (transformable) robot is versatile and effec- 
tive in space applications, entertainment, natural disasters, and developing and 
controlling such robots is really challengeable [1]. Exist mechanism are attempt- 
ing to connect their modules as external parts, as well as try to maintain their 
structural shape as snake or walker [2][3]. Always challengeable to design the mo- 
tion mechanism and autonomous capability for such kind of novel architectural 
artifact which is entirely different from the existing. 


2 Lecture Notes in Computer Science: Authors’ Instructions 


In this video, we presented a transformable artifact (COLUMN) which be- 
comes a social mediator in the proposed context. The COLUMN is a soccer 
ball-shaped interactive artifact consisting of eight modules that are connected 
to the twelve servomotors (Figure 1). We can transfer the COLUMN body shape 
by moving the actuators with modules, and these actuators can be controlled 
by a wireless communicator called a ” COLUMN Gear.” Each of the users (three 
participants) can control the 4 servo-motors of the COLUMN, and a user can 
swing the COLUMN gear to change its body shape (transfer its modules). Users 
have to coordinate with each other to boost its transformable behaviors while 
preserve their co-action in the dynamic interactions. In contrast to the previous 
utilization of robot as a social mediator [4][5], in this study interactive users 
have to understand what /how they are controlling the robot, and necessitate in- 
ferring the way of connecting with others, and also understand essential in-time 
coordinate in the dynamic interaction. This type of synchronization with each 
other’s swing (toward achieving their goal) can be defined as the interpersonal 
coordination in the context. Therefore, by using the patterns of interpersonal 
coordination, our motivation is to explore users invent transformable behaviors 
for the COLUMN. 
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Extended Abstract 


Machines are often displacers. In factories, they displace human workers; in construction projects, 
they displace natural objects to make way for buildings and roads; in communication, they displace previous 
means of social interaction. Machines are often big and powerful. Such attributes are necessary in order to 
move heavy and/or large objects in order to reshape the landscape. Some may interpret these activities as 
machines overpowering parts of the land. Machines are often noisy. This quality, in part, defines the sonic 
identity of contemporary city life and has been reflected in art, for example in George Antheil’s composition 
Ballet Méechanique (1924). 


Drum Circle challenges the idea of machines as displacers that are big, powerful, and noisy. Here, the 
robots, CADI (modular percussion arms) and MADI (a snare drum with fifteen strikers), depart from their 
familiar man-made surroundings and venture into the woods of Virginia. The robots tuck themselves 
peacefully into the landscape, arranging themselves on top of logs and covering themselves with leaves. Once 
comfortable, they sonify their physical surroundings by playing a diverse percussion ensemble that includes 
found nearby objects, such as beer bottles, as well as more traditional instruments such as woodblocks, metal 
bowls and drums. In some sense, this contextualization of machine in nature strikes us as a juxtaposition of 
unlike things. Indeed, machine / nature interactions often result in dramatic transformations where nature is 
displaced and transformed into some unlike object(s) of human will. Drum Circle is not such a story: here, we 
see and hear robots not as imperialist amalgams of electromagnets and plastic, but rather as agents that are 
governed by the kinetic and acoustic characteristics of our! physical world that can cooperatively interact and 
coexist with surrounding objects. Their stature is small; their movements are subtle. Their materials are 
translucent, allowing images of their surroundings to pass through their bodies in recognizable but altered ways. 
Their utterances are quiet and, at times, sparse in order to allow us to hear nature’s responses. They 
communicate in the spirit of harmony, not hegemony. They present themselves in a way that invites us to 
question our assumptions about the line between nature and machine. 


Musically, the relationship between machine and nature in Drum Circle is one of collaboration: a 
mechanically regular input is filtered through a system characterized by natural components, physical forces, 
and pseudo-randomness to produce a temporal sequence that reflects a synthesis of these various elements. 
The piece begins with a sequence of falling rocks that is the product of mechanical regularity and physical 
randomness. One of the robotic percussion arms is attached to a box, filled with small rocks, with a hole on 
the bottom of it. The arm shakes the box at regular intervals, which reconfigures the rocks and allows some to 
fall through the hole. The rocks strike a sensor-equipped metal plate and the temporal intervals between these 
strikes are stored in a software program. The resultant rhythm from this sequence of intervals is iteratively 
transformed rhythmically and orchestrationally throughout sections of the work, thus this statement and its 
means of production are conceptually and musically foundational. Over the course of the work, these musical 
ideas are stated, absorbed, re-interpreted and stated again to create a cyclic yet developing story. 


Drum Circle was composed in 2010 by Scott Barton and Steven Kemper for the robotic instruments MADI 
(multi-Mallet Automatic Drumming Instrument), CADI (Configurable Automatic Drumming Instrument) and 
assorted found percussion instruments. MADI and CADI were designed and built by Expressive Machines 


1 Our’ in a shared sense: not a possessive one. 


Musical Instruments (EMMI), founded by Troy Rogers, Steven Kemper and Scott Barton. A video of the 
work is featured on the 2011 EcoSono DVD Agents Against Agency. 
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Abstract. 


This paper considers the use of tele-operated and autonomous robots in live 
performance. The word “robot” traces its lineage to theatre, and theatre has 
proved a conducive site for studying what makes robots compelling. Theatre is 
a narrowly defined domain in which robots can excel; however automated per- 
formances that merely substitute robotic actors for human ones are not always 
engaging. While some plays explore ambivalence to robots or “misbehaving 
machines” thematically (like R.U.R.), the exigencies of live theatre (unlike film) 
do not allow for editing or special effects. This means that robots must be high- 
ly calibrated and run the risk of appearing like over-rehearsed actors. How do 
artists create engaging performances while ensuring reliable and robust perfor- 
mances? 


This paper considers design and approaches to staging robots in live theatre 
and interactive art. Citing examples of machinic performances absent of hu- 
man actors (Le Cour de Miracles; Stifter’s Dinge), interactive robotic art 
(Telegarden; Six Robots Named Paul), human-robot opera and musicals (Death 
and the Powers; King Kong), puppetry (Pygmalion Project; Savanna; How to 
Train Your Dragon), and traditional spoken-word plays (I, Worker; Sayonara; 
All in the Timing; Fremtiden), we demonstrate how creative approaches to ro- 
bot dramaturgy and puppetry-inspired control techniques create compelling 
and interactive performances. Theatre performances function as authentic 
sites of human-robot interaction staged in fictional landscapes that both exag- 
gerate and occlude the capabilities of robots. 


Keywords. Art Robots Performance Uncanny 


